feat: add high-performance HNSW RaBitQ index - #1798

RaBitQuantizer::dense_layout (RaBitQuantizer.h:53) changes the on-disk code layout but is not part of the standard wire format: index_write.cpp:1027 tags every multi-bit IndexRaBitQ as Ixrr and index_read.cpp:2856 default-constructs the index, so a faiss::write_index/read_index round-trip restores dense_layout=false and reinterprets dense codes with packed offsets. read_RaBitQuantizer does not recompute code_size, and for any d % 8 == 0 (64, 128, 768, 1536) the dense and packed sizes are byte-identical, so nothing rejects the file — every distance is silently wrong. IndexRaBitQFastScan::IndexRaBitQFastScan(const IndexRaBitQ&, int) (IndexRaBitQFastScan.cpp:106) copies the quantizer and then reads sign bits at orig_code[j/8] >> (j%8), which is garbage for dense codes, also without a guard.

Fixed. Dense multi-bit IndexRaBitQ now uses the distinct Ixrd tag; the standard reader restores dense_layout, recomputes and validates code_size, and validates the code buffer length. Converting a dense index to IndexRaBitQFastScan now throws explicitly. Cross-reader round-trip tests cover the standard and cppcontrib readers.

Every assertion in test_hnsw_rabitq.cc is self-referential: CheckValidKnnResult (:65) only checks id range and isfinite, CheckKnnOrder (:78) only checks monotonicity, the serialization round-trip (:274) compares the index against itself, CalcDistByIDs (:253) compares two paths that resolve to the same distance computer, and the dense-vs-packed case (:282) compares two layouts of the same algorithm. If the query were rotated with a different matrix than the base vectors, or the shared RaBitQ distance formula had a sign error, all five TEST_CASEs still pass and the index ships returning wrong neighbors. HNSW_RABITQ appears in no other test file, so this suite is the only coverage.

Fixed. The test now computes an independent FP32 brute-force oracle and checks recall for L2/IP, RBQ1/RBQ4/RBQ8, and aligned/unaligned dimensions. It also verifies COSINE storage/refine distances against original vectors, so rotation, sign, or metric mistakes are no longer self-validating.

tests/faiss/CMakeLists.txt builds faiss_tests from a hardcoded FAISS_TEST_SRCS list that does not contain test_rabitq_simd.cpp, and this PR does not add it, so the 174 lines of new AVX2/AVX-512 comparison tests never run in knowhere CI. The only remaining coverage for the ~1400 new kernel lines is the dense-vs-packed case in test_hnsw_rabitq.cc:282, which exercises exactly one runtime-detected SIMD level: on an AVX-512 CI host the AVX2 dense kernels are never executed, yet they are the ones that run on AVX2-only production hosts, where a kernel bug produces wrong distances undetected.

Fixed. test_rabitq_simd.cpp is now listed in tests/faiss/CMakeLists.txt. I built faiss_tests locally and ran 10 RaBitQ tests, including explicit AVX2/AVX-512 single and batch-4 equivalence and the non-BMI2 fallback.

IndexHNSWRaBitQ(IndexPreTransform*, int) delegates to IndexHNSW(storage, M) (IndexHNSW.cpp:241), which only builds an empty hnsw(M); entry_point stays -1 so HNSW.cpp:1006 returns immediately and every search yields no results, while the constructor still sets ntotal from the storage and presents the index as populated — and add() is overridden to throw, so the graph can never be filled in. It also sets own_fields = true at line 19 before calling validate_storage() at line 22: if validation throws, ~IndexHNSW runs and deletes the storage the caller still owns, giving a double free. Neither this constructor nor the non-const pretransform_index()/rabitq_index() overloads have any caller.

Fixed. The unused pointer constructor and the unused non-const accessors were removed, eliminating both the empty-graph behavior and the exception-path ownership hazard.

this should not be here. It needs to be in index_read.cpp and index_write.cpp. Yes, let it be duplicate. Also, please make sure that our four-CC codes do not intersect with Faiss ones. So, a given four-CC should not be used by Faiss.

Fixed. Knowhere HNSW RaBitQ serialization is now implemented in the standard Faiss index_read.cpp/index_write.cpp, while the cppcontrib implementation remains duplicated as requested. The tags are IHRK, IHRC, and IRKC; I checked them against the bundled/upstream Faiss tag set. Cross-reader tests cover standard Faiss and cppcontrib IO.

read_knowhere_HNSW (index_read.cpp:1309-1320) omits the validate_HNSW(hnsw) call that read_HNSW performs at :1306, and the IHRK/IHRC branch at :2552 only calls validate_storage()/validate_cosine_storage(), neither of which inspects hnsw.levels, hnsw.offsets, hnsw.neighbors or hnsw.entry_point. This is the only HNSW-graph-carrying fourcc in read_index_up that bypasses validate_HNSW — the branches at :2611, :3464 and :3488 all go through read_HNSW. Take a valid HNSW_RABITQ index serialized by knowhere (fourcc IHRK), set one entry of neighbors[] to ntotal, and load it through faiss::read_index — reachable from src/index/faiss/faiss.cc:345 (BinarySet bytes) and :365/:368 (file), and exercised directly by tests/ut/test_hnsw_rabitq.cc:323. Deserialization succeeds, then the first search reads out of bounds, because HNSW::neighbor_range (cppcontrib/knowhere/impl/HNSW.cpp:53-58) indexes offsets[]/neighbors[] with no bounds check.

tests/faiss/CMakeLists.txt:10 adds test_rabitq_simd.cpp to FAISS_TEST_SRCS, which feeds both the if(__X86_64) target at :38 and the if(__AARCH64) target at :61. The file unconditionally instantiates AVX2/AVX512 explicit specializations (:502, :509, :516, :524-529, :539, :546, :552 are new in this PR) whose definitions live only in rabitq_avx2.cpp/rabitq_avx512.cpp, which cmake/libs/libfaiss.cmake:75 and :124 strip from FAISS_SRCS on aarch64. Building on aarch64 with -DWITH_FAISS_TESTS=ON fails to link faiss_tests with undefined reference to selected_float_sum<(SIMDLevel)1>, multibit::compute_inner_product<(SIMDLevel)2>, multibit::compute_inner_product_batch_4<(SIMDLevel)1>, rearrange_bit_planes<(SIMDLevel)1> and bitwise_and_dot_product_with_popcount<(SIMDLevel)1>. Reproduced on aarch64 with g++ 11.4.0. WITH_FAISS_TESTS defaults to OFF (CMakeLists.txt:42), so CI does not catch it.

The new IRKC branch at index_read.cpp:2385-2392 reads the transform count with only an nt >= 0 check, while the IxPT branch it was copied from adds FAISS_CHECK_DESERIALIZATION_LOOP_LIMIT at :2410 and validates chain[0]->d_in == d, chain[i]->d_in == chain[i-1]->d_out and chain[nt-1]->d_out == index->d at :2421-2443. validate_norms() at :2394 only checks inverse-norm count and positivity, so a top-level IRKC index gets no chain validation at all. Load a crafted IRKC index whose single chain transform has d_in != d through faiss::read_index (reachable from src/index/faiss/faiss.cc:345). Deserialization succeeds, then IndexPreTransform::apply_chain transforms with mismatched dimensions during search. A large nt additionally drives unbounded chain growth that the loop-limit macro — used 13 times elsewhere in this same file — exists to bound.